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Reporting Data with “Over-the-Counter” Data Analysis Supports 
Improves Educators’ Data Analyses 


Abstract: The benefits of making data-informed decisions to improve learning rely on educators 
correctly interpreting given data. Many educators routinely misinterpret data, even at districts with 
proactive support for data use. The tool most educators use for data analyses, which is an 
information technology data system or its reports, typically reports data without guidance 
concerning the data’s proper analysis. A solution to data misinterpretation lies in applying medical 
labeling conventions to information technology to essentially offer educators over-the-counter data, 
meaning reports are paired with straightforward verbiage on the proper interpretation of contents. 
Findings from a quantitative study involving 211 educators of varied backgrounds and roles at nine 
schools throughout California included improvements to respondents’ data analysis accuracy by 
205%-436% when one of varied forms of data analysis guidance was embedded within the 
information technology reporting environment. Findings were significant and can be used to 
improve data systems and data use. 


Introduction and Theoretical Framework 

Over-the-counter medication is deemed negligent if not accompanied by textual guidance proven to 
improve its use (DeWalt 2010). No or poor medication labels have resulted in many errors and tragedy, as people 
are left with no way to know how to use the contents wisely (Brown -Brumfield & DeLeon 2010). Labeling 
conventions such as those used for over-the-counter medication can translate to improved understanding and use of 
non-medication products, as well (Hampton 2007; Qin et al. 2011). 

Thus, in the way over-the-counter medicine’s proper use is communicated with a thorough label and added 
documentation, an information technology system used to generate student data can include components to help 
users better comprehend the data it contains. Yet such data systems currently display data for educators without 
sufficient support to use their contents - data - wisely (Cobum, Honig, & Stein 2009; Data Quality Campaign 
[DQC] 2009, 2011; Goodman & Hambleton 2004; National Forum on Education Statistics 2011). 

A data system is a computer system meant to provide educators with student data to help solve educational 
problems (Wayman, 2005). Examples of data systems include student information systems (SISs), assessment 
systems, instructional management systems (IMSs), and data- warehousing systems, but distinctions between 
different types of data systems are blurring as these separate information technology systems begin to serve more of 
the same functions (Bill and Melinda Gates Foundation, 2007). 

Labeling and tools within data systems to assist analyses are uncommon, even though most educators 
analyze data while unaccompanied by someone who can help them use the data appropriately (U.S. Department of 
Education Office of Planning, Evaluation and Policy Development [USDEOPEPD] 2009). Essentially, data systems 
do not commonly present data in an “over-the-counter” format that offers educators guidance in the report contents’ 
use. For educators, whose primary purpose for using data is to treat students, using data without embedded support 
is akin to using medicine from an unmarked or marginally marked container. This study considered the impact on 
educators’ data analyses if data systems provided data in an over-the-counter format, meaning data usage guidance 
was embedded within the reporting environment. 

Problem 

Educators make data analysis errors impacting students, yet data systems do not include analysis help, and 
it was undecided whether adding supports to data systems can reduce the number of analysis errors. Data-informed 
decisions can lead to improved learning (Sabbah 2011; Underwood, Zapata-Rivera, & VanWinkle 2010; 
Wohlstetter, Datnow, & Park 2008). Most educators have access to data systems to generate and analyze score 
reports (Aarons 2009; Herbert 2011). Educators worldwide are expected to use these reports to make decisions that 
impact student learning (Hattie & Brown 2008). 



Unfortunately, educators do not use this data correctly, and there is evidence many users of data system 
reports have trouble understanding the data (Hattie 2010; National Research Council 2001; Wayman, Snodgrass 
Rangel, Jimerson, & Cho 2010; Zwick et al. 2008). For example, in two national studies of districts known for 
strong data use, teachers’ accuracy when interpreting data was only 48% correct and it is unlikely other school 
districts would perform any better (USDEOPEPD, 2009, 2011). 

Few teacher preparation programs cover topics like assessment data literacy (Halpin & Cauthen 2011), 
most people analyzing data received no training to do so (DQC 2009; Few 2008), and human biases compromise 
judgment and complicate decision-making processes (Kahneman 2011). Even when professional development and 
added staffing are employed to improve data use, these supports require added resources and - though generally 
beneficial - are not foolproof. For example, in a study where teachers received PD in measurement, all teachers 
struggled afterwards with statistical terms and measurement concepts (Zapata-Rivera & VanWinkle, 2010). Also, 
knowledge management research indicated knowledge is hard to share with others, even when the intention to share 
it is there, especially when power or status is involved (Cho & Wayman, 2009). 

Data use impacts students, and misunderstandings when using data systems can cripple data use in school 
districts (Wayman, Cho, & Shaw 2009). Yet labeling and tools within data systems to assist analysis are uncommon, 
even though most educators analyze data alone (USDEOPEPD 2009). There is a clear need for research identifying 
how reports can better facilitate correct interpretations by its users (Goodman & Hambleton 2004; Hattie 2010). The 
power of data systems that generate these reports will not be realized until researchers contribute to improving data 
system design to improve analysis (DQC 2011). 


Methods and Purpose 

The purpose of the experimental, quantitative study was to facilitate causal inferences concerning the 
degree to which including different forms of data usage guidance within a data system reporting environment can 
improve educators’ understanding of the data contents, much like including different forms of usage guidance with 
over-the-counter medication is needed to improve use of contents. The study’s primary independent variables 
included the following types of data analysis guidance, each of which was framed in two different formats and was 
used with two reports per study participant to answer four data analysis -based questions of varied complexity: 

• Footer: A report footer is a brief set of text at the bottom of a report that communicates information an 
educator would need to know to correctly understand and analyze that particular report’s data. The 
study’s footers ranged from 34-58 words, 156-269 characters without spaces, and 224-324 characters 
with spaces. Footers were either monochromatic or contained minimal color used purposefully; for 
example, “Warning” was featured in red and “What to Do” was featured in green. 

• Reference Sheet: A report-specific reference sheets, also called an abstract, is a single page that 
accompanies a report to help the educator more easily understand the report and analyze its data. The 
study’s reference sheets contained the report’s title, description, image, focus (content reported), and 
warning (vital, cautionary information an educator would need to avoid the most common analysis 
errors made when analyzing the particular data being displayed). Half of the study’s sheets also 
communicated the report’s purpose (key questions the report will help answer) and additional focus 
information (intended audience, and format in which data is reported). 

• Reference Guide: A reference guide, also called an interpretation guide, is a 2- or 3 -page reference 
guide that accompanies a report to help the educator more easily use the report and analyze its data. 

The study’s guides adhered to either of two formats: (a) the report’s reference sheet (as described 
above) functioned as the guide’s 1st page, and pages followed containing the report’s instructions (how 
to read the report), essential questions (showing the user where to look on this report - and what to 
look for - to answer each question listed in the purpose area of the guide’s 1st page), and a “more info” 
section (offering where to get additional information on related topics) ; or (b) the guide contained the 



report’s title, description, warning, essential questions, and a “more info” section (details for these 
sections was provided earlier in article). 

The dependent variable was accuracy of data analysis -based responses, measured by a survey containing data 
analysis questions. 211 elementary and secondary educators throughout California answered these questions while 
viewing one of seven report sets of student data (Fig. 1-7). 


REPORT 1 


Grade 7 English-Language Arts CST Performance 



Worn Analysis and Peadng Literary Response wmtsn Wnting Writing 

'Vocaouvy Comprehension and Analysis Conventions strategies Applications 


RE PORT 2 


Students' CELDT Performance 

(Perfoimance Level in Each Domain and Overall) 



Figure 1: Scenario 1 Participant ( Control Group ) Handouts; No Supports Were Received 



Figure 2: Scenario 2 (Footer A) Participant Handouts; Note Footer/Support at Bottom of Page 
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Figure 3: Scenario 3 (Footer B) Participant Handouts; Note Footer/Support at Bottom of Page 







Figure 4: Scenario 4 Participant ( Reference Sheet A) Handouts; Participants Also Received Figure 1 Handouts 



Figure 5: Scenario 5 Participant ( Reference Sheet B) Handouts; Participants Also Received Figure 1 Handouts 
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Figure 6: Scenario 6 Participant ( Reference Guide A) Handouts; Participants Also Received Figure 1 Handouts 






Figure 7: Scenario 7 Participant ( Reference Guide B) Handouts; Participants Also Received Figure 1 Handouts 


Findings 

All supports used in the study had a significant, positive impact on educators’ data analysis accuracy. 

Educators’ data analyses were: 

• 307% more accurate (with a 23 percentage point difference) when a footer was present and 336% more 
accurate (with a 26 percentage point difference) when respondents specifically indicated having used 
the footer, 

• 205% more accurate (with a 12 percentage point difference) when a reference sheet was present and 
300% more accurate (with a 22 percentage point difference) when respondents specifically indicated 
having used the reference sheet, 

• 273% more accurate (with a 19 percentage point difference) when an 

• reference guide was present and 436% more accurate (with a 37 percentage point difference) when 
respondents specifically indicated having used the guide, and 



• 264% more accurate (with an 18 percentage point difference) when any one of the three supports was 
present and 355% more accurate (with a 28 percentage point difference) when respondents specifically 
indicated having used the support. 

On average, the 21 1 study participants indicated they used supports 58% of the time: 

• Respondents receiving footers indicated they used them 73% of the time, on average. 

• Respondents receiving reference sheets indicated they used them 50% of the time, on average. 

• Respondents receiving reference sheets indicated they used them 52% of the time, on average. 

87% of participants who receive no supports indicated they would have used footers, reference sheets, or reference 
guides if the supports had been available. 

When no supports were used, data analysis accuracy was 11%. All 211 participants, regardless of support 
use, averaged a data analysis accuracy of 26%. In cases where respondents indicated they used an available support, 
data analysis accuracy was 39%. See Fig. 8 for visual representation. 
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Figure 8: Impact of Supports in Terms of Relative Difference 


Significance 


The findings of this study filled a gap in education field literature by containing evidence that can be used 
to identify how data systems can help increase educators’ data analysis accuracy by providing analysis support 
directly within the information technology and its reports. The study also rendered examples and templates for real- 
world implementation, which are available for free, open access, on the researcher’s website. Improvements data 
system and report providers make in light of this study have potential to improve the accuracy with which educators 
analyze the data generated by their data systems. This improvement will likely benefit students impacted by 
educators’ data-informed decision-making. 
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